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SOME APPLIED RESEARCH .CONCERNS « 
USING MULTIPLE LINEAR /REGRESSION ANAljYSIS -' 



Introdiaction 



During the last fifteen year 6, multipljb linear regres- 

siori, tl>e general ca^e of /the le^st squarnei - solution, has 

developed, i^tc^ a "^OTiriq^ate/ researfch. technidbe for the 

sociajL scie^ices. With tliis increase in tifiage ^f- multiple 
^ ^ ^ / ' * / 

linear regression,, there have fleveloped opposing view- 

7 / . 

pointfS with regard to its usefyilneSs and/ its arppropriate-^ 
ness,, Tli^ arguments of botli <^roups, the/ advocates and the 
crdtics of multi^ple linear regression, can be found in the" 
t^ftcent literature • ^ The advocates of mi/xfiple linear 
regression state end defend mjl^advantkges ^ovided to the 
researcher who uses multipl^^ linear regression. The 



critics s,tate a variety of iimitationte and concerns with 

^ ^ J r r 

'respect to utilizin^vmultiple linear /regression as a 

/ / ■ ' 

research technique. ' -"l ^ 

. . I ^ / . 

The purpose of this *ijlaper i's to examine' the advantages 
clailtied by the advocates 6f multiol^ linear regression and 
some df the concerns expr/essed by iJtiS critics. More than 
anything else what the authors of phis paper have attempt- 
ed to prov^ide is an ovSc^ll refere/nce on how a researcher 
can ^apply rau^iple linear. regressj[on in order to utilize 



antages that- it has to oflex. Also, the authors 
.tempted to provide a nuftiber of meaningful and 
cal methods by which rekearchers can deal with the,:, 
rns that are often citef^ by the critics of multiple 
r regression, which are ^correlation/causation^ upward 
1 and multicollinearity . • . 

Advantages of\ Using Multiple Linear' Regression 

While a great deal of money, and time is currently \ 

.ing directed toward reseai;ch, there appears to be -a 

eneral lack of acceptance of the relevance of research 

findings. One reason for .the present skepticism has been 

that th.e statistical models used by researchers have * ^ 

frequently been unrelated or tan^entially related to the * ' 

research question of interest. There are a variety of ; 

^eaisons foi* this. lack of agreement betyeen the research ' 

question o5 interest and the statistical model. 

* * / 

One such reason is that courses that 'teach research " 

methods generally emphasize, data-' analysis-, rather thaw 

j5racticing appropriate methods ahd procedures for asklhg 

an<3 developing research questions.' These courses do not 

aldequately develop the skills of evaliaating the rese^ch 

questfon and the statistical ' modeifi that are most capable 

^of reflecting ,thfe^research questions . , / . 



Quite often, ';a student cpming 6ut of these coiWses, 



tends to select a f ami liar ,| "canned" standard statis>tic|l^^ 
design, or package (cookbool^ approach) such as a 2 x 3, or 



• * ; * . 3 . 

< • 

t \ • . 

2x2x^3, because he has not been taught to develop his 
' Q^jnj^models^to reflect* their research question. Therefore, 
he uses therse, standard models which dictate the questio^i 
\ ^ being 'investigates!. Sometimes a researcher ij^' aware that 
these models .do not completely representees trfc^ 'Research 
question. In addition, a . significant F-value^on a factori- 
^ ^'1, design is often difficult bo interpret. When this* * 
happensr 'he may then make inf^erential jumps from his data. 
These inferences may well be inap{)ropriate. ^ 

Therefore, in many cases the researcher is unaware that hi 
models are not really reflective of his research quest4^ns;' 
and quite .often/ the unsophisticated researcher allows the 
'statistical model to totally dictate hj.s research question. 
Under these conditions, we find research- that is techni- ^ 
cally. correct but is not relevant because it is not 

r * 

relatfed in a pragmatic way to a specific, problem.. (Newman, 
et ah , 1976) * ^ • • 

,1. One advantage of using regression procedures is 
that th\e' re'keatcher finds.it necessary to. first, state his . • 
hyiDo^thesis anA ttien write the regression model ndec^ed to . 
^ test that hypothesis. Thus, every t^st of significance is 
directly testing a ^specif ic question po^ed by the research-^ 
er. Also., regression is^more flexible in allowing the* 
resea<rchef to write -th^e models that specif Ically, "reflect 
" his question of ihterest. The advantages provided by this ' 
flexibility can.be seen ih research questions that deal ' . 
with' interaction variables, .directional and partial 
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interaction covariance, trend analysis,' and questions that 
encpunter the problem of di*sproportionality - 

2. In dealing with interaction variables, a research 
er with^'regression can ask interaction guestiotns between 
.catagoriaal' variables, between catagorical and contimious 
variables or between continuous variables. Since 
regressidn can deal with catagorical and continuous vari-. 
ables, It is more flexible in its ability to reflect real- 
world problems than oth^r statistical procedures. With 
regression, there is ncy'^need to catagori'ze variables that 
are continuous in. naftiure as required, for example, by 
,traditit)nal ANOVA; therefore,^ one would not lose/ degrees 
of freedom 'and power. (McNeil, Kellyf McNeil, 1975, 
Kerlinger, 1973) ).\ , . / 

An example of how a hypo.thesis which involves -the 

group irtembership co^il^d be tested is lisW^l below: ' ' ^ 

/ \ " ' . ' ^ * 

Example 1: ✓ / ' \ ' ' ' 

♦ 

^ posttest score • 
X]_ = control gro,up '\ 
»2'^ experimental group 
X3 = I.Q. .score ^ / 

^4 " ^1 * ^ (I.O. scolres for tjie. students dn the 
control group) ' ' 

X5 X2 *'^X3 (I.Q'. scores for the students in the 
experimerital gfQxip) * , ^ 

••^1,2 " .error >^f6r*^each .subject r , 

U = the uni*^ vector * • 

^^\' * • ' " -partial regression coefficients^ 



^'Rp' = 'variance in 'accounted for by tKe full'inodel 

to '^' \ ' ' 

= variance in Y accounted^ for. by the restricted 
* ^nodel ' ^\ ^ ^ 

dfrt^^'^.the number of JLinearly independ^ent vectors in the 
^ / ' model minus the "^niimber' of 'linearly dependent 

'vectors in the ^restricte^ model. • . . ' 

df*^>=.t'he nximber of subjects minus -the number of 

linearly independent vectors in the full model. 

=^the differences* between the posttest scores 
of the control gr9up and the experimental group 
are not constant across the range of I.Q. scores.' 



Model 1 

Y 
I 



Restrictions : 



34 = = 33 



Model 2 • = -a^U + a^X;^ + ^3X2 + 33X3 + ^2 

By tJlting Model 1 against Model 2, tl^at is, by deterrain- 
Jing if the F-value calculated by: . ' 

F = ' ■ 

■•(i - R|)/a£, . . _ 

is* significant, the researcher could dete;rm^ne if there' 
is a significant interaction ^etween the cbnti^iuous 
va^iable ^of I.Q. scores ^ and the ^categorical variables/^ 
of the groups (McNeil, et al-^, 1975). ^ 

3: Regression also allows the researcher to tesf 
direct! ojjel and partial interaction questions (McNeil, 
at ai^, 1975). For example, t;he researcher may hypothe- 
size that* I.Q. scores have a greater Impact on the ' 



posttest^ scores' of the subjecis 'in " the experimental group 

than it does ^or the cdi^trol group subjects. The 

researcher could obtain, the -answer t6 his .research questdon. 

by. testing Model 1 against^ Model 2 .(using tl^e same variab- ' 
" ' ^' • •■<♦,,' ^ 

•les and models li-sted previously)'. If a significant ^" 

J- ' ' . 

F-yalue was obtained and if .ag a4, the research could [. 

conclude that I.Q. scores ^had a greatier impact on-' post- 

test- scores for'the subjects of ^.experiifierital group than 

for the subjects of the control group. ' . * , 

Regression also a:]flow$ the research to test inter-' ' 

action questions that the researcher would* tejid not to 

ask if he was^not familiar with^ regression procedures,^ 

that is, partial interaction questions' (McNeil, et ^I., 

1975). For example, a researcher might be interested in 

* # ■ 

testing the following .-hypothesis (Fraas, , 197-7) : ' 

Exampld 2 : ^ , ^ * ^ . 

H = Previous economic training has a greater impact 

on the- avQ;rage ^>os€'^est'* s'cor.es of the students, in 
the two experimental groiips than for '.students in 
the. two control groups. (Note: More than ..€5/0* 
groups co\ll^ be used.) 
= posttest scores , \ ' 

= previous economic training (I if y'esl^ 0 *other- ' ' 
wise) * * . 

^f^' no previous economic training. (1 .if yesj 0 other-' 
^ * wise) ' ' ^ 

X3 = Control Group I (1 if yes; 0 otherwise) ^ 
^ Experimental Group I (1 if .ye's; 0 otherwise) 



■r 



?6 



- -^7 



X, 



j= Control Group' ll..(r- i£ ye's; o otherwise) 

= Experimental Group. II (1 if yes; 0 Otherwise)-'' 

• « ^ • 

=^ ^1 * ^3 Students iij- Control GroujD I with 

previous- eeonomic training (1 ,if' Ves; 
. ^ 0 otherwise) ' . , ' - • . 

^ ^1 * ^4- Students in. Experimental Group. I^with 
* ^ previous econdmic training (1 if y.es; 
. 0 otherwise) , v • 



10 



= ^1^* ^5 Stul^ents 'in Control Group II with ' t 

previous . economijd.trainijig '{:l 'if yes; 
0 otherwise) ' ;y • . 

= Xj^ *• ^6 •Students in Experimental Group II with 
previous economic training (1 if yes; 
0 obherwis^) 



X 



11 



X 



12 



" ^2 * '^4 



X 



13 



^14 



Students* iji; Control Group I wi^h no ' 
previous, economic training (1 if .yes; 
0 othe;rwise) 

Student, in Experimental Group I with no 
• previous economic trainihg (1 if *yes; 
" , 0 otherwise) 

= * X5 Students in. Control Group II ^with no 
previous economic training, (1 yes; 
^ O otherwi-se) - A / * - # 



= *X2 * X 



g Students in. Exper ^.mental Group II with ' 
no previous " economic training. (1 if yes; 
0 otherwise) 



^15 




Xg .+ X.7 


^16 




Xg - X7 


^17 




^10 "^ ^^7 


^18 




^ii ^7 


'^19 




r^l2 " ^7 






^13 ^7 


==21' 
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Variables need to impose • the 
restriction- required to test 
the hypotheses. ^ • . 



J 
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/' 



-8' 



t. \ 



Model 1 % / . • ^ * . • 

" -: ■ V ' • • - , • 

■ • , a,,X, , + a,-X,^ +. a,,}; + a X + E- 
p - li li 12 12 ■» ^13 13 14 14 

/2 = • 



Restrictions: - j^^?-^ ^9) ~ (^8 ^lo) 

^ (T^i-i " - 612 " ^ J 

V . . "19^19-^ ^20^20- -^-^1^1^^ ' ' • 



.■ If *the researcher firtds a- significant T-value and^if 

the. value "of the left. side of . tlfe • restriction is greater 

than tlie value of- the right side of the restriction., the 

• • . ■ . ■ ^ \ - 

researpher would conclujie that the data 'supports the \ 

hypothesis. Witt;out the J^nowl-edge, of regression, the 

-researcher njay not even ask such a question^ - le^t alorie 

be able Jto test it, even though the question may be_^of 

g^reat-' importance to his study. 

4. A fourth advantage of regression, is. that by ^ 

using the multiple linear regression procedures, questfpns 

-that involve covariance are easier to test. and interpret 

(Kerlinger, 1973; Kerlinger and Pedhauzer, "1973; Ward and 

■Jennings, 1973; williainis * 1974; Draper 'and Smith, 1966; f 

Newman, et al., 1976; McNeil, •1976) . .This point can be' 

" demonstrated by the, procedure listed below: . 

ERIC ^ . . . . ■ • . 



Example 3: * ^ * 

Hypothesis I: 'Ttie polsttest scores fdr the .experimental 
group 'aife significantly ^higher than the' 
4 • ^ postt^3t, scores for the control group over 

and above the differences due to I.Q. 
. scores. * (The ' variables^^isted In Example. 1 
. are also used for this example) . ^ ■ 



Model 3- 



'^1 ^*^o" ^1^1 +^^2X2-^+ a^X^ 



Restricti*oa: aj_ a 



2 



Model 4 



Y^'= agU + .a3X3 + E 



f 



If Model 3 is found to be Significantly diff-erapt "from 
Model 4, this would indicate that there is a significant 
^differencd between the groups/, Also, if ^ an < a / this 
would suggest thaj: the Experimerital Group had higher ' 
pdsttfest scores than did^ the Control Gro^p (at some 
specif icOO level) . - " ' * ^ , 

5. Another^ advantage of regression is that*' it facili-' 
tates the calculatioa and ^irfterpretation of trends (func- ' 
tional relationships). When the research question of^ inter- 
est is one of trends or functional relationships , one often 
finds the .use of inapfJropriate statistica^mod^s .which 
cannot accurately reflect 'the research , questions tNewman, ^ 

'When- researching developtnental questions, on^ is often 

^. ^ ' 

more interested in functional relation^hios tharf^ mean 

^. ^ ^ 4 ^ - 

differences. There is generally a continuous variable' that . 



is Of interest, /sQcb ^iS.time, age/ population siz-es^-I^.O, 
When trMit^oi\al: analysis of variance -is employed^/ for ' 

example, contintte^s'^variables are 'foraed into categoriza- 

. ^ • * ^ " ' • ~ * ,^ • 

ti^ns, T.l)is causes the researcher to 'lose degrees of 

' freedom, arid ther6 is ^a potenfelal Ibss^ of iofdrmatipn* . ' 

This loss, is contingent upo^**'how reprekent4*ive the ^ 

categories are^ of .the inflections in the naturally occuring 

contiiluous vaAablfe; ' , - ^ 

■ • / • ' * - ' ' :^ 

Since continuoi^wariables are' frequently artificially 

categorized,^ the .analy^s produced .tiv such ^ proc-edure', raay^, 

v» . ^ ^ ' ^ * 

not reali^ reflect the researcher '*s' question of interest. 
* . . " • (. ^ ^ ^ 

The most efficient method for* writing statistical- models 

that reflect trend or curve fitting question,?,, is*, the. ^ 

general <:ase of th^ lea'st squares, solution, linear model. 

(Multiple Linear Regression Procedure sV Newman (1974), . 

McNeil, Kelly, McNeil .(:i9 75) , Draper ^ Smith (1966), Kelly, 

Newman, .and' McNeil (I97i)). This , procedure allows one to 

write linear models, which specif ically- reflect the 

research question, v ^ i < - ^ * 

, Linear Regres-sion is an- excelfent statistical tool for 

looljiing at ^a population trind or ^rofnparing , multiple trends ' 

over time (Newman (1974), Erying. (1975)^. '.''''^ ^ " * ' 

For Examt)le) in ^Figure* 1, a graph is pr^se»t)ed that" 

a?eflects t!tie researcher's interest^ in learning ^ if , there* are 

significant differences in trends (in tjiis case slope 

differences) between, subjects who received, a Developmental 
f * ' ^ " • . 

Reading grogram^ (Xn)^an^ students who "did not receive ^the 



Program . (Xj) f ■ as it relat'es to their cumulativ.e G.VtA. . 

* Example 4: . * . ^ ' 

•Hypothesis I: There are significant differences in sXopee- 
' , ; 'for X and predicting the student's 

' cumulative G.P.A. * " \ "« 

* The models needed t^ test thi sN^^r^thesi^^^e .as' 

follows:: ' . ! ' • ^ * , 'wKlflF 

Model r '^ = agU + a^X^ •+ a^X^ + a^X^ ^ • 

\ Restriction: a-,' = a, 



^■4 



Model 2 Y, =v a^U +''a,XT + a.X. + a^X_ + E - ^ 

J 1 U 1122 55 ^ 



^ = eumul'itiv^ G^P^A, • \ » ' . . * * ' 

X = I'^if student had program, 0 ^otherwise 

= strident did not have pjf0^^m, ^0 other- 

wise * T 

X^ = nloftiber of tHe quarter hours f or ^ th6 , srCb-- 
j^ts^ who had thp progi;am, 0 otherwise 

X = number of the quarter, hours for the* sub-* 
'•' ' ^ jects who .did not have the program^ 
;' ' 0 otherwise ' r - ' - ^ 

*^ / X ~ ^3 ^4 " number of quarter hours'' for all 
; ' ? . ' subje(jts . \ ' - 

* * U = unit vector, 1 if subject u:s in the sample, 

O otherwise ' ' . 

. ac- ^ partial regression weight 



/ 
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.FIGURE 1 ,1 

TREATMENTS AND CUMMULATIVE G.P.A. FOR^-STUDENTS 



DEVZLOPMESfTAL % 
READOG PROGRAM 



CU>MJLATIVE 



G.P.A* 




. (3 • 20 tfO '60; 80 ' 100 - 120 ,140- 160 180 * 

QUARTER ^mS * 

^ • V - \ %. 

If Model 1 is foungl to be significantly different f rpm ^ 
Model 2, that is, -the F-value' is significant, this would * ' 

* v. 

indicate that there is a significant difference between 

students who took the program and students wh^ did not take » 

•t^Tis^program in terms of their cumulative G.P.A. 

" • ^j->xBegressi will also allow many otkefe: ^questions to be 

asked when dealing with trend analysis. Second ^degree or 
• • • " < 

third degree rela.tionship.s (curvilinear relationships) 
could be investigated. Regression models cou Id ^ be written 
that would reflect such trends. 



1 



*'6. The applied statistician an^ researcher is 

\ • • ^ * • , 

.plagued with the problem of disproportional cell sizes in" 

• ^ - 

' factorial' experimental designs.^ This ma^ pccur because of 

mortality in the laboratory animals being used^n tWte* 

experiment; thej required number of subjects 'not available;, 

someone who had agreed to ta'ke part in the experiment fails 

to show up; or ^he data may represent the piropof tion^lity 

that exists in the "real wo^aJrd. " (Ifewman,- Or^ec2rl977) 

When tha researchers feel disproportionality is severe 
enough^ to be of concern, there hre a variety of procedure:^ 
.that he 'can utilize to attempt to correct for the" potential 
problems. Ho.wever, bef or^ any ^rr«ctions ar^ applied/ 
op^ should be sensitive to the underlying assuipntion that 
they are making about the population from which their data * 
is dt^wn, and the investigator must also be very clear 
about the research question he is^ inter(4sted in asking. 

The following is a check list adopted from Newman and 
Oravecz (1977) , of the type of inforrnation that a resear/ch- 
er should investigate before selecting a ftiethdd for correct- 
ing for disproportionality: ' ' 

a. know -something about phe theoretical and/or* 
empirical relationship between the vari£tbles being .studied; 

b; know some of the descriptive data about the - 
population ^ne wishes to generalize to in relation to the 
specific Variables being studied; 

' c'^ know the specific research question under investi- 
gation if" one deqides an adjustmejit for disproportionality 



is needed, then * , 

d*^ know the underlying assumptions ahd^mplications 
for differfent^ adiustmentii| procedures, aijd' : 

e. kn^y^ the^ consequences for using the Selected ^ 
'adjustment proce4ure ^ph the interpret^ation and generaliza- 
tion o^ the dar^a. 

A detailed, disotj^ioii of the unditl^ing assinnptions 
can be*found in thfe article*.by Newman and Ofavecz (1977). 

^ Tbere are a variety of solut^ns to ,the "uaeguar- N • s 
problem, which can be divided into' two major categdries — 
approximate and exact. 

> • * 

Exampless^f approximate solutions are^ randomly 

eiiminatj-ng data >and runndngz-the analysis on just' group ^ 

\ . * ■ ' . > ^ ir 

means, the'ref ore,' decreasing thd numljer and power., A 

researcher usiag" any of theg^e solutions "i*s generally aware 

of the limitation's fiftid problems. ^ * ' ' 

IThat" may be more misleading are the exact, solution^ 



which are all technically correct but vffiich,' like the niean 



> 



median, and mode, are Answering different questions. ,The 
three exact solutions, which are listed below," - are th4*" - 
full rank method, • fitting constants method, aj\d the , 
hierarchial ilfethod {Newinan/& Newman, 1975).. ^ 
Example 5: ^ . , • ; ' 

'-A. * Solution 1 - Fuir Rank Solution 
A symbolid' exaittple of • this -.procedure is presdnt^d 
belbw. for a two factorial desigp. ' . 



Model 1 



Model 2 



Model 3 



Model 4 



15 



'kab 



= .'6 f b OL + bj^e^ + b,a3_^ + e 



kab 



kab , 



2'^b 3 ab 



kab 



+ b€aa + 



*4bgcr^ + 



^^ab+ ^kab 
"^^kab'. 



"^kab scoie for subject k in row a and , , 

column b ^ i * ' 



. 6 =/i* the grand X 



\ ■ 



7, is the Effect for r-ow "a" 



-Pb 
ab 



= is J:he effect for cQlumn -"b" 

• 4 

% • 

= is the interaction effect for the row "a", 
and coXumn "b" i 



^kab the' error ^ term for^ach subject 



b. 



. b are partial regression coefficients 
n . ' 



Adjustment foT- 



ution 1 



Adjustment for A main effects test ModeL 1 against 
Model 2 . ' 

Adjustment for B main effects test Mod^l 1 against 
Model 3 " ' ^ ^ \ ^ 

. Adj^ustment for A*B effects^ test Model 1 against . 
Model 4 . . . . 
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The following "is a symbolit: -representation of this 
solution: ' - . 

Adjustment for Solution 2 . ' - . ^ 

5?^^ W ^ ^ : ^10«a ^A ,-^ ^kab " » 

55°^^" '^[kab = ^I2^b.-^^kab ' * ' * 

Model 6 . Y ' ' = 5 + bn ,a + e 
—• • k^b' ^ . 13 a kab 

Adjustjn^nt for .A , main effects test Model 4 against 

Slodel 5 ,; , . * ' ■' ■ ^ , 

, • Adjustment - for B main effects test I^odeL 4 against 

Model 6„ « . ■ - 

. AdjustBjielSt to± .AB interaoii on^ef ^Cts^est Model 4 
, . ' ■ f — -jf—- 

againpt Model 1 , 



* Solution i - Hierarphial' Method (Cohen (1968), 

WiiaXams (1974 r 



T^ae following is a symbolic representatioh of this 
solucion:- .„ . .... 
Adjustmerit for goliftioA 3 • ^ 

^ ' . 'VT 

Model 17 Y ' =" 6 + b, a + ^ 
- *kab 14 a" kab 



Model 8 = 5 -k^^c, , * 
kab ; kab 

Model 9 " Yi;,K = 6 + b,^a + b,' B + e 



kab - - "IS"' - "le'' " ^kab 

^ • " . ■ .18 
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Adjustmeht for A main eFfects test 'Model 7 against 

Model 8 ^ . . * ' , . 

- i ' / ' ' . " 

* Adjufstment for B. main effects te^t Model 9 against 

Model .7' _ '< / . \ 

' * •' • ''*'»' • ^ , ■ , . 

Adjustment for AB- intera<;^ion test- Model 1' against 



Model 9- 



A, 



•4 



Each of the three leag^t* square solutions make dif £er-'> 
. . . , '</ • ' - ' ' ' - ^ " - 

'/ent assuiAption^ about* the meaninqfulhess and "usefulne&s" ' 

• ^ of* the cprrelations between the* A main effect, B main 

effect, and AB interaction.^^ ' ' • 

^ , Solution,!, 'for . exaiji^'leV ji^hen testing the'^A 'main 
effects 'as sumea , the correlatiCn between A and B and the AB' 
Intera'ctiCn^ i;S of an acci4^nt4l^* nature / and therefore 
should not be |::onfeidered '(Rock,;et ^al. I'STS) . This , • 
. solution is most likely to be pr^fereid when Wae can assujJe^^ 
I th^t5 the missing subjects pro<2ucing' di^proportTonality 
. were rando(n. .If one is finable ^ m^Re this, assumption 
'then.it woiird be Inappropriate to 'lise Solution 1, (which* 
may be the cag^ most* frequently) / ^ " , • 

'go^lurtion 2 assiames that therfe is, no correlation 
between the* A and B "l^in , effects in 'the population. There- 
fore, the correlation betiS^een A and B in .th^ sample is * ^ ^ 
.a functioti of disproportionality and not representative of 
,the population. ' .^iutiOn 2. ^^hen attempts to adjust for 
- this correlation^ 

However^ ^Solution X assumes that the corre^atioas 
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betwe'eh thj| main' etf feci ts and ciie iijteracticJrij whioJ^re-' 
9ults from .the dispropor^tionality, ar4 not r spuribuS and 
are ch-aracteristic of the popQlatipn. Therefore, it does 
not attempt to ad j list for this- correlation. 

, ' . • 

If one cannot assume that the^ correlations between* 
the A and B m^in effects due to dispr^ortionality are, ^' 
dije t9 chance, -then Solution 2 would be' an inappropriate • 
correction.* ^ . 

, Solutioft 3 requires ,:an , a priori ordeting of the 

.importance of ; each variable. Let us assiame thatithe 

a prion tjrdering ar^: A main '.effects, B main effects, 

■ ■ , ■ . . - ■ • ■ ♦ / 

AB interaction, respectiv^ely. (Nevmian and Oraveci; 1977) . / 

It 'is imptirtah't to dgt&rmirie which' of thes^ methods- 

> ^ : — > : — , 

are reflecting Jbhe 'question that we are interested in •» 
answering . ^ One can only do this by bping sensitive to 
one's research question ancj.b J being, aw^re of the differ- 
ent* statistical techniques wl\i^ are more .appropriate 
than others. i • - 

/ Methods • That Can Be Utilized ^ 

J , T6 Deal With The, Concerns-. Of ■ I ' 

~ I ' '~r~ ■ ' ■ 

Correlation, Upward Bias R^^alues, And Multicollinearity 
• There are three concerns which have been Qxpr^ssed 
by the -critics of multiple lijiear regression that have 
drawn a gteat deal of attention. One concern expressed^y 

fel V \ > 

some critics is that causality cannot be .inferred from 
studies that use * regression procedures. Another concern . 
that ha^Sf^een expressed, by some researcher ^.s the 



tendency for multiple correrations tOi.be upward bias. * 
The third major concern, called multicollinearity 1^' ' 
problems produeed by the non-orthogonality of th« inde- 
'^pendent variables. '[Note:' One of the problems wi£li 

li 4 * • * 

* » / 

. disproportionality, a concept discussed in the "preceding 
section 6f this paper/ is that disproportionality- produces 
correlation between its variables, i.e. ,/mu;LtiQOllinearity. ] 

^, * It. is tjie purpose of- this section of ^the paper to ' 
present a discussion of possible methods -that a reseajrcher 

'couljd 'use in ^rder t,o deal with these prolHams. ' - 

'^1^ One of the concer^is th^ has bee,n expres'sed 'by > 

^he oritics of multiply linear r^grtession is that one 
canhot' infer causation if regression or correlation is 
used. This concern whiqh hast)q.q^n expressed bo^h f^ally and inr 
fonrvally , can Se found* in a recent article entitled "Regression • 
Analyses and .Education Production Function^': Can They 5e 
Tru?tfed?" The authors Lyecke apd McGinn (1975) conclude 
that a^ researcher cannot appropriately ifrfeS: causation 
^rom regression techniques*. 

V* The statement by Lye^^e and McGinn (i975) is/correct 
Howev^, causation dannot be inferred ^from an^-^^atistical , 
tool unless an 5\ppropriate research design is utilized. 
If causation is to be inferred;, regression as a statistical 
tool', as is the case for any qther statistical tool, must i 
be used ia relationship with some research desig'h that 
can be found, example, in Stanley and Campbell (1969). . 



♦ - , 

To the Extent that -this design has internal » 
validity, the-xe searched can infer causal relationships 

> 

between the independent variables and dependent yariables. 

If a research design is ek post' facto, where the 
independent variable is not urider the control of the 
researchers;^ no matter what technique is used, one, cannot 



infer causation. l£ is not. technicalALy 'legitimate to 
infer .causation ^/hen the design is ex po^t fac^to. Eveff 
though a ^variety of statistical techniques such as path 
, analysis as developed by Blalock (1962, 1964, 1970, and 

( N 

1972) and more- recently component analysis developed by 
Mood (1971) , have attempted to get at causal relation- 
ships ot^ ex post facto data, through the manipulation of 
' ^ ^regression techniques, one s;biil cannot technically infer 
. cassation (Newman & Newman, '1975>. Newman and Newman' , 

stated the following with 'regard to^ caugation and 

# •( * • 

- ' component analysis: ^-^^^ 
I 

Since one of the major purposes for calcu-* 
lating component analysis ds to .attempt to 
^ improve the explanation ofVex post facto ' ^ ' ' 

re'search designs, this can lead^one to mistaken- ' 
* ly believe that the unique v^|btnce accounted 
for by aji inflependent variabl^R/ith a criterion 
.is of a causal' nature (p. 45) . 

In a similar fashion, Lee Wolfle (1977) , states th^ 

inabil4tyW a researcher who is using path analysis on 

ex post' facto, data to infer causality as follows: 

k Although path analysis is a method for con- 
} . sidering cause, • neither it, nor any other meth- ' 

od, can be used for inferring cau^lity from 
non-experimental data (pr. 39) 



erIc ' . \2t 
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.It is, therefore, not the use of multiple linear 

regression that .precludes the researcher from inferin 

. ^ ' 

causal relationships between the ' variables. It is tj^e ^ 

lack of a true'^'eSfperimental design that prevents 'the 

researcher from making such inferences, ^ausation^ can 

only be inf ei:red if a true experimental design was 

utilized, ifregardles of the statistical tools that were " 

used tt analyse the data. » * 

2. Many researchers mistak^nly-^lifeve it is 

meaningful to include in thei^^ reports only that a 

manipulation of^an independent^ variable was shown to have 

a significant 'effect upon a dependent Variable. The 

magnitude of this effect is not given^to the reader. 

THe magnitude 'Of this effect, which could be presented 

by citing the or values, must be taken 'into con- 

sideration v/hen a researcher is interpreting the practical 

significance o^ experiment results (Byrne, 1974, 

Cohen, 1969, *?redinan, 1972) . 

Most researchers ajre avare that a R-^value tends to be 

'ijigher in the sample than in the population from which the 

sample was drawn. This ; shrinkage is due to the fact 

that the regression weights are ^calculated to maximize^ ' 

■^the prediction'df the -criterion. ^^-The' sampling error is 

•capitalized on i^en^ calculating the regression weights, 

so that the predictive power for *any ^one sample is 

maximized. It should be noted, however, that in an 

article by Dalton (1977) it was suggested that this 
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overestimation of R"^ may not be too great in many cases 
and is really a good estimate of the population value. 

Dalton (1977) used ~ Mottte-eaf-l-o methods to compare 
2^ r2 .(r2 a^ter a shrinkage, formula has been applied)', and 

2- * * 2 * ^ 

^ . The bias in R was consistently positive and it 

decreased as th^ sifciple size increased.. However,- Dalton 
ccfncluded frdm his study that even^ though* R^ and 2)^ were 
superior to R""* when n < 30, ^*R^ showed little 'feias in^ large 
samples*. ' Therefore, this ^study^may suggest that the v • 
upward bias tendency for R"^ values is not as prominent * 
^ problem as pnce thought. . HoweVer, there was one short- 
comming of^Daltion's (1977) study and that is he only ex- , 
amined at'the thr^e variable situation. This gq^tly 
limits the possible generalizablity of the study* ^ 
/ There arh four possible methods which can he \ised 
to obtain a corrected R^ (R ) • These methods are 
entitled the -Wherry Method, McNamara Method, Lord Method, 
-and Cross-validation Method. Uhl land Eisenberg (1970) 
empirically investigated the Accuracy of three of these 
methods; Wherry's original formula (1931), McNemar's > 
modification (1962), and Lord's (1950) formula. These 



formulas^ar^^: 



R' 


= 1 - 


(1 


- r2) 












=^ 1 - 


(1 


- 




= 1-= 


(1 


- R^ 



N-1 • . 

N-K . (Wherry), 

jj^^lj^ • (McNemar) 



N+K+1 
N-K-1 
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(Lord) 



/ 

where: R = the corrected estimate of th'e miLltiple m 
correlation < 

R - the' actual calculated multiple cArrelation 

K = the number 'of independent variables*' 

. 7 the number of iiidependeiit * obsetvationsr 

^Jihik and Eisenberg found that even though Wherry's and ' * 
McNemar's 'formula^ are the most commonly used. Lord's- 
formula consistently gave more accutat^ estimates for 
tlie five different N*si2es they investigated. (N = 50, lOD/ 

150; 250, '325) and for the situations using two througln:^^^^ 

) ' 

thirteen predictor variables. 

A study conducted by Klein and Newman (1974) indi- 
cated that \fhen there as;^e 100 subjects for each 
* variable all three formu-li produce thd same estimates. 
When the^ratio is less than that, Lord's formula is con-' 
sistently more conservative, that is, it shrinks more. 
As' the variables increase, there seems to be a tendency 



f or _ McNemar an^J Wherrj| 1*^ produce more similar results. * 
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.In a discussion between Keith McNeil and Isador.e 
Newman, the topic of the ratio between variables and sub- 
jects was reviewed. McNeil stated' that this ratio may not 
be equivalent for , continuous variables and dichotomous J 
variables.' McNeil suggested that in order to establislrtr 
a. 10:1 ratio for a continuous variable, one may have to^ 
have ten subjects per "grouping," of the variable, that is, 
if 'the value's, of a continuous variable distribute them- 
selves into approximately thre^- distinct groups , ten sub- 
jects ar^e needed for each group in order to retain a 10:1 . 
ratio. This is probably a conservative estiritgite, but ^there 
is no data to empirically support the claim of equivalent^, 
ratio for. continuous and dichotomous variables. 



.Klein. and Newman further stated, that .it, is conceotu- 
ally meaningless to interpret negative R ^ and; since ttie 
low^t possibly one can legitimately .obtain is 0, it ' 
sterns that t^hese formuli need a cotrection factor added 
^o that th:ey are bounded on the low And by. !r2 =; ,o . 0 and 
on the high end by R^ = 1.0. It .is therefore suggested 
'triat if orne uses any of these^ three shrinkage estimates 
tfiat any negative R^ be interpreted as if it were R^ = 0. ' 

Ketl/, et al. (1969), suggest cross validation' 
p^ocec^ie^^ as estiraj^tes of *shrinka<^e instead o^^sjlng the 
more ^mathematical approacfies used by T^erry, McNemat, and 
•Lord. The cross-validation procedure estimates the 
shrinkage by applying the weighting coefficients from the- 
original sai^le to a new sample of subjects from the same 
population. 

For example, assume the weights for ;iodel 1 in 
Example 3 are as follows: 
Example 6':> 

Model 1 ^ 'Y^^^lOU + 6.85 + -5.00X2^'+ .05 X^ + E ' 

^ A neV sample stiould be taken from the same population 
and the variable Xg (the predicted criterion) sholild be 
generatedvfor this sample by using the weights obtained 
from the first sample. The transformation, needed would be 
as follows: j . ' * 

Xg = 6.85 * + 5.d"0 * + .05 * + 10 

* * * 
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- If the correlation between (the predicted criterion) 

and Cthe observed -criterion) was qa|^high as the R-value 

^ * ♦ " . • • ' ; * * n9 

for Model ti)e researcher could consider th'e R -value 

for Model l**to be, stable/ , \ \ ^ 

Some of the dif fe2;|gpc,es in\he shrinkage estimates/ 

using, the different, jSrocedur^s amy be explainable. For 

example, Wherry'fe and MCNemar ' s . formulas* berth, attempt* to 

estimate the population R, based on the, sample, whila 

Lord's fopnula attempts to estimate the R from the sample 

to' antoher sample. ^ T^is is conceptually similar to the 

Cross Validation procedure suggested by'Jtelly. In deciding 

which method of .estimating shrinkage is to be used, it^^is 

important to consider the underlying, assumptions of each 

procedure (Klein,, and Newman, 1973). That is,' cross 



valid^tion will *end'to be, more conservative estimation. 
Thus, it wil^> tiyid to' -produce^ larger shrinkage in R . If 
one is. interested in\making predictidns based on one sample 
to* another 'Sample, cro\s validation and Lord's approach.. 
J:end to 'be the better estimates, however, if • one wants - 
to estimate pbpulatiqn .values from a sample, T^Therry'a and 
"^ctf.emar 's^approaclxes would be preferable^ " ^ \ 



3. Thd third major concern, which chas drawn a great 
deal of attention, is multicollinearity , that is/ a situa- 
tion in which the predictcz^r variables are nonorthogonal. 
One 'of the problems that multicollinearity can cause ies 
large standard err,o^s in the sampling distributions of the 
standardized regressipn coefficients. These large standard 
errors allow small changes in the relationships between 
independent variables from sample to sample to produce 
. large regression weight differences' even though their 
signs tend to be stable.. TheVef ore,^=*''i^terpreting regres- 
sion weights can be highly misleading due to this high- 
variability (McNeil, et al.^ 1975). Another problem 
caused by -multicollinearity is that a researcher is more 
likely to -coijimi.ttee a T^e II error • (Vasen & Elmore;^ - 
1975). " V ' . • I 

There. are a number of ways to deal' with the problem ' 
of multicollinearity. ^ive such methods are: 

a. 'component regression , ' * 

b. factor regression 

ci ridge regression , , . * 

dj "benxgn neglect" ( • ' 

a System of equations. 



e 

% III 



A. One method suggested in .the literature for 



dealing with multicollinearity is component analysis 

(Newman and Newman, 1975; Massy, 1965). Component analysis 

is a procedure vrhich divides variance into .two proportions; 

Unique variance (Uq) is* the proport;ion of variance ♦ * ^ 

« , ' j> 

attributed to a particular va^j.able when entered last into 

the regression equatioR. Cotranbn variance (Cv) may be"" 
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conQeptUally thougjit of as the^egree of overla]^ of 
correlated variables in th^^-^ediction of' the' criterion. 

Any given common variance must be independent of unique 

If 

and other common variance. 

The calculation of unique variance for three predictor 
variables could be handled' as follows*: /■ 
Example 7: 

Let ' \ • . » ' 

f 

T , 1 

grade point' average . - 

Xj^ = SAT score 
= I.Q. score 

= high school class work 

The number of independent components 'can be calculated* 
by the equation: ' . ' 

• 2^ -1 • 

where: N = the nximber of predictor variables'^ 
Tl\us, for this exainple, the- number of independent 
cpmpor^nts would-be equal ta: . 

' 2^. -1 7 , . ' - ■ ; 

The- -number of sets Qf unique variance is, equal to the 
number of predictive .variablps. For this example, there ^ 
would be thijee sets o^ uh'ique variance [Uq (1), Uq«'(2), 
uq (3)]*. The ^number of sec6nd, . third, etc., order variance 
can be dfetermined*^ by the following fortnula:' 



where: »N = number of predictor variables ' 

" -n p number of variajiles taken at a time 

NCj^. =' number of cjombinations erf N objects,^' 
^ taking rt ,number at a ti^ne/ independent 
* • ^ of order. 

• • - -l 

In. this example, thyTiumber of second and third' ^ 
otdef commonalities are 'equal to the following: 

_ -3! _ _ 

""■n • 2! 



NC = — = Mr - 3! ' 

n- 21 (3-2). '"'^ 3! (3-3) ! " ' 



The thftse sets of second order commonality' are 
^^(1,2,), ^X(l,3)' CV(2',3,)'' and the third order comraon- 
ality , variance is CV(j^2,3)- » ^ 

These Components are additive "and when summed , 
will equal the total proportion of v&riance accounted for 
by thp r| of the '^ull model. Mood (1969) developed a 
rule for determining the Rg necessary for caluclating^ 
unique and common components of variance. The rule is' 
to develop products of -the variables being considered. 

For exam'ple,/ if one i^ interested in the Xiq,[yi-^ in 

•this exkmple^with three "predictor variables (X ' X , X,), 
■ . ( , 1 2 »3 

'first subtract thaf^ variable of interest (X ) from one, ■ 

multiplied -b,y a -1,- and multiple other variables in the 

' / ' 

equation. ' • 

mile: -Kl-X, ) X,, X, = 
> • . 1^3 

Next, take the variables that are a product of the 

2 

expansion. aAd calculate -the Rg that is indicated by each 



set ^(separated by + and - sigfTs) • ' ^ 

■L y y . < - 

V 

In a similar manner, cme of the second order and the 

t 

third order commonality vjiriaftc.es -woXild be' calculated as 
■follows: . " ' ' • • 

• rule:- -I{l-Xj;^ (1-^2^ ^3 ' ' 
-X3 + Xj^X3 + X2X3 - X^X2X3 ^ 

\.. • \ ' ' ^ 

.rule: -l(i-X^| ^^"^2^ (I-X3) = 

-1 + X^. + X2 -X1X2 + X3 - X1X3 - X2X3 + 5(^X2X3 



fNote: ^When a .one is by itself in the expansion, it is* 
ignored in determining which R| should be calculated.]** 

For further details in how to calculate component 

• -1 . . , - , ♦ 

•^analysis, see -Moo*- (1969, *971) , KerTinger (1973) and 

Houston and Holding (1975) . 

\ ' 

With/all technique^/ one must be aware of the ' ' 

liitilfeationS' So th^t .the technique* can be employe^»most * # 

efficiently. The fo*llowing are some of the limitations 

one' should *fce sensitive to when using component analy^sis 

(Newman and Newmart (1975)) : 

1. As in the example, when there are three predictor 

variables, there will be seven components.,. One can eis'ily- 

2 * 

see th,e rather large number of R s that have to be 'calcu7 
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lated for. just three predictor variables in the full model., 
HoW^^r,^ in usin^ multiple regression, the investigator ^ 
frequently ""has many more than #three predictor variables. 
Therefore, the humbei^ .of conjapnents can easily become 
impractical, to handle. ^ i ^ 

2. . An i;itegral part of component analysis is the 
concept of U<^. Uq .is operationally defined as:^ 

^ variarfce accounted for by a variable Wheii^ 
* entered last in a multiple regression equa- ^ 
tiohr ^ , ^ . . ^ 

Therefore, the Uq depends upon and- is affected by the 
variables that are already under investigatics^. Even^ 
•though the .Uq independent, in the set of variables 
for that sample > the variable is not independent. 

^ • * I -J ' 

3. . the number of predictor variables increase, 

the number- o^ higher order commonality components al^b ' 
^increase.. Just as.it is difficult to interpret higher 
than ,thir^ order interactions in traditional analysis . 
of variance, it is also difficult to interpret higher than 
third orde^r commonalities. 

4. .In examining some df the formuli for calculating 
the comittenality components, one becomes sensitive to the 
possibiAity that some of the coirtponents can e'asily accpunt 

Cifor a negative, proportion 'Of variance. When this 



situation is encountered, it becqmes very difficult to^ 
interpret or make conceptual sense out of the analysis. 

5. , yiSod '(1971) stated an' important limitation one 
should consider. The unique variance ^(Uqf* accovmted for by 
art independent variable <^:an change radically from situation 
to situation. However, the attributed a factor that 
the variable is a part of is not likely to change. There-> 



Mc^od. suggests that one should group the variable^ 

^ • .1 ^ ' . 
based on the underlying concept thev seem to be measuring. 

This 'would produce a more stable - estimate. This^ group 

process will also have • a .>side benefit of rekofiing- the 

total number of predictor variables which will make the 

component analysis much more manageable. However, if one \ 

uses the procedure suggested by Mood, the weighting of 

each variable * becomes a problem.. Do^ the factors account 

for the same 100 percent o^ the proportion of variance 

accounted for when each variable is used separately? J If 

not, one is loosing possibly* significant information. 

Fina^y, it is f icult to decide on which variables shouid 

go together. Quite often, variables that, look as if they 

are measuring th6 sai^ie underlying constrUct,^ are not. 

III., B. .Another methcpd by which a reseai^-Cher can deal 
with the problems caused by nonorthogonal predYctor 
variables is factor regression. 

Factor multiple regression is a procedure that may ^ 
circumvent, sonte of the problems associated with component^ , 
regression (Massy (1965), Duff, Houston aijd Bloom^ (1971), 
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Gonnett, Houston and Sh^w C1572) , Uewm^n ;i*972) ) . It is' ' 

, a -method th^e enables one to empirically, determine the 

.factors with which* the ^^|!^iables are associated.- 

. . * The first step in the procedure is to orthogonally 

factot a set- of independent variables into a nXn factot • 

^matrix. Connett, et al. (1972) suggests that this factor 

matrix may be rotiated, but only with a rotation thdt 

preserves the opKiogonality of the factors.' The next 

* step is to standardize the independent variables. This 

matrix of standardized variables is ^ostmultiplied by the 

matrix to obtain the factor vari:^bles. Because these 

t^toT variables are orthogonal, the beta weights cf these' 

^ , variables, when used in a regressiqn equation, will tend 

to be stable. Therefore, this procedure allows greater 

.interpretation of the beta weights^to be made. ^ 

I An additional advantage. of using factor sc6res is 

that when a matrix is factored much^of the eifror variance 

tends to be distributed in the factors that account' for 

t 

.: ' the^ least variance. Therefore, one of ^ the possible by- 
products of using factor scores which account for mott 
trace variance as predictors is the • likelihood 'of in- ' 
' creasing reliability; therefore, decreasing shrinkage 
(Newman, 1972). ^ ■ ' i 

If one is interested in improving the multiple 
regression equation by urtng factor techniques, there is 
. only one way this can be done. That is, .the number of 
m ¥ faptors used must be les^ than the number of original 
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•' variables. This will "increase the df and also possibly ' ) 

decrease shrinkage-estimates. Because' o*f thi$, feome 

\ * ' • 

' researchers ^have used only the .few f'actorJ* that account 

for the "greatest" amount of the factored trace. However, 



\ 



" when this is done^ one may be los^ing information that 
can afccOunt fpt criterion^ variance by eliminating a 
factor that accounts for very little trace of thie 



irs ror very little trace 
itVs highly correicTCed wi 



factored matrix but \is highly corref^ted with the 
criterion scores. 

Using only the factors that account ^f or most of the 
trace should be avoided when the prdictor variables that 
are being factored are likely to be highly i^liable. Some 
examples of such variables are: height/ weight, rel^^on, 
sex, income, age, etc. ^^^dhder these conditions, a variable 
that accounts for little of the trace variance may be 
a good and^ highly reliable predictor 6f criterion 
variance. 

When using factor regression orre should be aware of 
when it can be most appropriately used. It is the 
-authors* opinion that the factor regression approach may 
• be more appropriate than corapoilent analysis when one is 



interes^ted in determining/'^e unique variance acco\inted 
^or, especially when the number of predictor variables is 
relatively large and there are a minimurri of ten- subjects 
^for ev^ry variable.. / However,^ if one is interested in the 
' commonality, the factor regression procedure is not 
appropriate, in tjAs /case, if one l^as a large number .of 
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... - ■ f • • 

variables' and subjects, it is possible to use factor 

• " " \ \ ■ 

analysis with oblique lyo tat ion. This procedure will 



y 
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condense the large number variables into factors 
which can be used .as a new ^et of predictor variables. 
Sincfe .these factors may be oblique (cotreiated) , one may 
then wish to do a component analysis w*hich^ "will yield - 
estimates of theNanique and common variance attributed 
to the factors. Obviously, the oblique solutions' lack 
many of the desirable characteristics which make the 
orthogonal solution easier to interpret. However, there 
are times when «i'.researcher~~rft2^be interested in the ^ 
common proportion 'of variance attributed to factors 
which are theoretically and empirically related. ' ■ r 

III. A method called ridge regression has been 



proposed- as cN^ossible means by which a res^ajcher can 
. obtain stable regression coefficients (Hoerl (1962), Hoerl 
and Kehnara (1970 (a), 1970 (b) ) / Marquardt and Snee * > 
(1975)). The ridge regreq_sii5# procedure requires that 
a constant bfe repeatedly addfed to the diagonal of the 
X X matrix (where the X variables are scaled so\ that 

S 4. X^X has the A^rm of a correlation matrix*) befpre the 

... ' ' 

matrix is inverted. That is/ consider the standard ipodel 

• for mult . linear regression ^ ^ 

Y ^^"^"^ ^ K ' ' * ' 

t X = nXp matrix -of p predictor variables at each 

. n d^ta points 

Y = vector of obsei/^ed- values ' ^ ( 

mc . . . 



» • * 

6 = pXl vector of population values of the para- 
meters' . ^ ^ 

=.nxl vectQr of experimental errors CE (e) = 0) 



where 



3 = CX^X)"^(x^Y) ^ 



X^X = the. product of transposed }^ and X • 
x/y the profluct of transppsed X and' Y 

A 

g = least 'Squares estimator of 6 

Ridge regression, as described in more detail xjy 
Hoerl (1962) and -Hoerl and Kennard (1970a, 1970b) is an 
estimation procedure based upon 



@ = (X^X + KI) -^(X^Y) - ' 

I = identity matrix 

K =: 0 < K < 1 * - 

A* 4 

3 = ridge estimator, of 
^ ^ ^ ' • * ' 

where K^is a( conststant number added to the- identity 
matrix^I. ^e researcher can determine thfe appropriate^ 
K value, i.e., the K value that stabilizea the regression 
coefficients by examining the Ridge Trace. The Ridge 
Trade is_^a plot of the coeffi-cient weigh1;s vs. the 
K values. A hypothetical dliagram of the Ridge Trade is 
7en in .figure 2 for the. variables X-]_, Y.^, and X3 . 



Regression 
' Coefficients 



FIGURE 

RIDGE TRACE 
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At the K-value where the ridge traces for the variables 

^appear to become approximaJtely parallel , 

the regression coefficients become ' stable . In 

Figure 2 the ridge traces become approximately parallel / 

where K = .,04. Thus, the researcher would use the 

regression coefficients that correspond to that p^nt. 

The researcher will find that for models! with low 

r2 values require larger values of K than do models with 

high R values. Also, increasing K indefin^N^y will 

ultimately force all coefficients to zero, l?ut it is no.t, 

• • • * ^ 

\ificoxnm%i to see a coefficient (usually after an initial 

si^n change) to increase in absolute value as K increases 

(Marquardt and Snee, 1975). 
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Before this procedure ' is used, however, a researcher 
should be aware o£ the differences between the coefficients 
produced by the least squares solution and coefficients 
produced b^ ri4ge .regression. The least squates solutiOip^ 
yields coefficients that minimise the residual sum of 
squares. The expected value of the coefficients* are j 
unbias (E (B) = 3) and have the minimum variance am'ong all 
linear unbiased estimators (see Figure '3a) . 

Jn^ridge regression, the variance of the coefficients 
decreases (see Figure 3b) as the value ^or K increases. 
Howey err , tke bias of this ^estimator in^r:eases (E (61 0) 
as the value of K increases. What the researcher is doing - 
with ridge regression isi accepting a little bias in the 
expected value of the cg^efficient in jjeturn for a lower 
mean square error [MSE = variance of the coefficient + 
(bias) ]. In fact, the objective of ridge regxession* 
to find a value of K which giVes a set of coefficients 
with sitjaller MSE than the one produced by the least 
squares solution. As the K value increases, the residual 
sum of squares will- increase . But remember, it is not the 
objective of ridge regression to obtain the "bei^t fit" for 
the sample data but rather to develop stable coefficients' 
(Marquardt, and Snee, 1975) . - , 
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F I 6 U R€ 3a 

Qfarquardt and Snee, 1975) 
VARIANCE AND BIAS IN AN ESTIMATOR 
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ERIC . 



Zero Bias 
.Large-Variance 
E (B) - 



3b 



> Non-zero ^ias 
S^iall Vfiriance 
4, (B) = b' 



The t)recdding discussion on the differences between 
the least ^squares solution and rid^e' regression ^dbes point 
out one limitation to using i^idge regression. Because \the 
espected vilues of the coefficient is bi,as, hypothesis' 
testing would.be questionable. Thus, if the researcher 
is attempting to test hypotheses, Vidge regression may not 
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be the correct .melfhod for 'hija' to use in* handle the problem- bf 
of multicblline.arity (unstable qefefficients) , 

4 ZTln .D; a fourth possible method for dealing • 

vith the problem, of mullicollinearity is not to deal 

, with i:^ The argument for thi^po^ion can' be 
'demonstrated by -the following example: ' * . 



♦ Example 8; 

Yj_ = 'post^st^ score^ 



i 



Xj^ ^ I.Q. score * '•■^ 

2 ^ * * 

X- = X- *»> X 

^1,2,3 ^^^o^ each, subject 
^.U '='unit vector ^ - 

^o* • -^2 ^^res,saon^oefficient weights 



Model 1 



Mc|d»l 2 



Model - 3- 



^1 " ^o" ■'-^l^i E 
^ - ' r- 



■A 



R 1- 
1 



■n- R' = 0, 

2 . 



= 0 



. Assume that the variable », is r-alated t6 .variable .' 
Y 'in 1»he m4nner indicated in Pigure 4. * 

' ; F^I G E , ^' S s 

THE RELATIONSHIP BETWEN' Xn AND * 



6^' 




/ 



FRIC *?°^^' E^cai^ae in Figuire'4 waS'glven by John Pohlman at the 1977 A-E,RhA« 

Convention tsa sxipport the argument^ ttiat-' forward Stepwise Regression^ 
may be elipin^ting sirtrpressor, variabaes, ; 41 



• "^^^ ^1 y^^^e for Model a would 'be equal one. . 'Hqw- 
ever, the value for Model 2 would 'Hf^l^'qual to zei^D fo^t ^ 

the relationship between and Y^., Also,' Model } would *' 

' ' 2 ' " 

^ave an R value equal to zero^r the relationship between"* 

X? ahd Y, . . \ ' 

Model^l,is attempting to account for the variance 

in Yj^ by using variables Xj_* and X^.. It is important to, '/ 

note that the' concelatitoh be^g[£ui X^-and- X^^ is high.^That is/ 

nJ^iticollinearity. is present £lB|lodel 1.* In Model 2 Jlbd ^ 

Model 3, the multicol^inearity is .^JLi'^iinated' by the , • 

tra.^i3r6naT pr(!>c$du^ of eliminating one of ^ the correlated 

independent variables^ What fras also been, -eliminated, how- 

ever, in both Model; 2. aind Model 3-. is a- surpressor varialDle , 

(Syirpressor Variables have also beei^ called in, intervening 

variable or a slaeperrvart^le) . 

||^A sutpressor * variable. rs present whfen a variable*- 

jias a law correlation Vith th^^riterion and is^lVighly 

correlated with some o%her 'variable in 'the^jppedictfive ^ \ 

Equation. ^la. addition, wM»n this variable i6. placed * in * 

rne predictive equation along with the vari^le -with which 

it is iiighly correlated, the R^ of thd predictive equatiojl ^ 

will increase' signif ijCantly^' ^uch^ a^*sLirpressQif varickbie ^ 

. r . \ y • ( I. ^ \- 2 ' > ' / ' ' 

is Bjfeseilt in Model 1^. When bpth Xi find Jf, / Variables which 

. ' ' ' ' V • . t> 

are highly correlated, ^re used together ds they are in 

Model 1, the R^-valu6 of Model 1 .increases J^igni^icantly 
oyer the f}^j^lyies of -Modfel 2 %nd Model 3. The point- is* ^ 
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that' the researcher does not waijt to eliminate a 
surpressor variable. 

Consider^ the followincf example: 
Example '8: - ^ 



Let.lfj^ achievement '.scores 
=5 treatment groups 



X2 = control group 
X3 ^= reaction time 
Assiime that is cbrtelated^th X^' and X^ is also 
correlated with X.. / ^ 

The hypothesi^s to be^ tested isVas fcllo^^i^rr' 

. 'H]_: Th^e'^is a • significant difference between^ 
. ^ - the axrhievement^ scores for the control^ group 
and the experimental group over and above 
the differences due to reaction time scores. 



Tjhe models belcwr 



1 



/ the hypothesis : 
Model 1 ^ * 



Restrictions: a^^ = a2 
Model 2 



+ ^2^ + 
f 
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It is important to nOtfe, however, that the 
.researcher must select his variables' carefully. That is, 
his hyf>othesis' should probably not include if the 
relationships between X-, stnd'the groups^ (Xn and X ') are 
not found in other research in the discipline br are illogical. 
If these relationships are not usually found or are unstable, 
the results of the hypotheses tests may yary ftoici sample 
to^ sample. . '\ ^ ^' , • 

The reBGar^:her should also be aware that he is more 
likely to commit -a Type IJ error when the relationship 
'between X^ and the cpoups iS not consistent across the ' 
contiilumn of YV That i^> , there is, an interaction between 
groups and the^, reaction- ifimeV" In- fact; when there is ?n 
interaction between groups an4 reaction tiijje, one o£ the 
conditions of coyarianice has Jjefen violated (hombgenuity ^ 

l^her^fore, ^iialysi^' of covariance »is no 
longer appropriate. • ■ ^ ' . " . 

III. Soper (1976) suggested in a review of a 

»study^ on the use of prqgrammed instruction in economics 
that* a Jsre^em of eql^iions should be 'establish^' in order 
to correVfe,^^ the nohorthogonality of' the iiraejendent 
varialDles. / For. examp^ei^ consider the following hypothesis 



H]_r There is a signiflc^nt^^f ference between 
the/ control 'group dnd i^jj^ experimental ^ 
group %in poSttest scores over' and above 
-the differenefe, due to scholastic ability. 



f4 



= posttest score 
•= pretest score ^ 



X2 =''sAT score (Scholastic Aptitude Test score) v 

= experimental group 
X '= control^ group 

m 

a . . .a = regression coefficient weights - * 

« 

^1* • iSrror terms (Y - Y) for the different models 

The :traditional method of analysying the data would be 

•9 * 

to test Model 1 against Model 2. 



Model 1 



Model 2 



^3 = n 



X, = a U + a-X. + a^X- -i-.E^ 
■L ,0 l-l- -.22 2 




If X^ an^^^ are correlated with X^^ (SAT scores) , 
Soper (1976)' wou^d suggest that a system of equations, 
or in, this case on .equation, would need* to be speci^fied. 
The needed equation would be as follows: ^ 

Model 3 - 



I 



^2 = ^o" ^3X3 ^4X4 + E3 ^ 



The vd\ue for would represi^nt the amoun-t of 

>. 

variation in SAT scores thafeHtre unrelated to group 
'membership. . * f ' ' . 



Next, ,E wpuiLd be us^ed as an independent' variables* in 
Model 4> and^M(3del-4 would be tested*against ModeL 5. 
Model 4 and' Model 5 are as foTlows: ^ ^ 



Mod|l 4 



4 



Restriction: ' a = a 

3 4 



Model 5 



Yi = a^U + a^^X^ + a^E:^ + 

However, the researcher has not ttested his original 
question of interest ilype VI error) which was: Is there ^ 
a si^ni^^ieant di'fferentre between the control gfoup and the 
experimental group on pos^^test s9ores, over and abov^ the 
differences due to scholastic ability? What he ha^, in 
fact, ^tested is the hypothesis: There is a significant ^ 
difference between the control group and the experimental 
group onposttest scores over and above the differences in 
I/Q. and SAT scores ^unrelated to group- membership. This, is 
adiffereni; questioni x 

Also, one must , be aware that correcting -numerous 
variables^ for multi^collinearity .tends to make the. interpre- 
tation of Vhe results very diff ici>lt. ^ The researcher may 
|iot *be able to practically ex{5lain. what a significant 
F-value' indicates. " ' * 

For exalffple, assume twp variables, I^Q. and sex, 
I were correlated and the# researcher set up a system of 
equations which included the followiAg equation: , * 
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4 

I.Q. *=t a U = a, sex + ' V ^ 

The question is. How do we -interpret E^? It i? whatever 
I.Q**is after sex has been removed fran it. * Is it still 
I-Q<?' Most pcoiDably not.*' ♦ ^ • ' " 

In cpncl^|ion, the authors hope that this presentation 
which dealt with some^of the currently ^identified p^^blems 
ifi conducting research, specif fically when using regression, 
has sensitized the applied researcher to these problems • 
and ajLternative solutions. The authors feel that no one 
paper can do justice to all the topics covered. However, we 
feel that this paper can be used as a guide to where one . 
may go for more detailed information. 

It should be kept in mind that the authors felt the, 
regression approach is probably the most flexible and useful 
single tool available to the researcher. However, like any 
other tool, it is only as good as the insights an^^ 
sensitivity of the user. IJo not commit the classical error" 
of asking a research 'question and testing it with a statis- 
tical .model that is incapable of ^ reflecting that question- * 
(Newman, et al.,.1967). ' * 
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